There exists a general need in the development of software/systems to ensure that the finished product is sufficiently tested prior to its release to the public. Such testing is performed to detect programming errors/flaws. Corrections are formulated and then incorporated into subsequent versions of the software. There are various levels of testing thoroughness. The more thoroughly the software is tested prior to release of the software to users, the less likely bugs will be exposed in the subsequent use of the released software.
An application program interface of an operating system comprises a set of methods defining entry points by applications into the functionality (e.g., retrieving/opening files, launching executable programs, printing, etc.) supported/provided by the operating system. Furthermore, the functionality of the operating system comprises groups of related functions referred to herein as components. In the Microsoft WINDOWS operating system, examples of components include: Kernel, GDI, User and Shell. Application program interface (API) testing ensures that operating system components (e.g., printing, shell, etc.), when called upon by application programs, execute in an intended manner.
One general technique for testing operating system components involves writing applications to directly call and test operating system components via their specified API methods. In such case, the application programs are written for the sole purpose of exercising the operating system component(s) under test. Such approach provides a high level of control over testing. By directly programming the calls themselves, a tester is able to tailor a test to include specific inputs to APIs under specific contexts. On the other hand, testing cannot be performed until the test program has been rendered.
Another general technique for testing an operating system API avoids the need to write a particular program to act as the caller to the components under test. This alternative approach involves running application programs and observing their operation (and errors/fault) with regard to the various called components of the operating system via the API. This approach avoids the need to author an application to submit calls to the operating system components. Furthermore, this approach ensures that at least the application used to perform the testing operates properly with regard to the particular operating system API. A potential shortcoming of this approach is the wide variety of applications that execute on an operating system and call upon its functionality through the API.
The above-described approaches to API testing, while offering particular advantages, also exhibit shortcomings including either an inability to exhaustively test an API or alternatively perform such exhaustive testing at a very high price in testing time and resources. One especially vexing problem arises during the exhaustive testing of an API either to verify proper operation (e.g., verify security and compare against expected results), during performance testing (e.g., determining machine cycles needed to complete), and during regression testing. In known systems, a test program is written and compiled into an executable that is then executed to exercise the API and verify returned results. Though this task is relatively easy to explain, carrying out the programming, if performed correctly, is time consuming. As a result, an inordinate amount of effort is devoted to programming the tests rather than defining its scope and specifying its content.